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TRANSACTIONS 

Cross Reference to Related Applications 

Q This application claims priority from provisional application number 60/267,643 filed 

S February 9, 2001 . 

Background of Invention 

**y [0001] Customer decision support over a channel often relies on searches. For example, a 
customer may make purchases based on searches done on a database. Many different 
^ online merchants have multiple different products for sale. A customer can access the 

i y online merchant's site and search for the product by name or characteristics. 

];!: [0002] For example, the customers may search by the word "battery" to find all batteries. 

Alternatively, the customer may narrow down the search by using more terms, for 
example "battery for brand X" to only find batteries of the type that are made for 
brand X. Typically the merchants return information about the product including 
product specifications and a picture of the product. 

[0003] This kind of searching can be carried out in many different client-server 

environments. For example, the large online merchants often allow the user to search 
in this way. The auction merchants, such as eBay, and the product comparing engines, 
such as "MySimon"aIso allows such searching. 

[0004] 

However, a user might not know enough about an object they desire in order to 
fashion an educated search, especially when the product descriptions are from 
multiple different sources. For example, a user may want to purchase a specified item 
or product that has a certain look. There may be 1 0 or even 1 00 different ways of 
describing this product. Just as an example, a winter coat made of leather could be 
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described as a coat, but could also be called a jacket, a parka, or many other different 
terms. Therefore, it may be difficult to adequately search the database, especially 
when the descriptions are produced by multiple sources. 

Summary of Invention 

[0005] The present application teaches a technique of searching a database over a 

channel using images and/or image information. In an embodiment, the database may 
be a database maintained at a server for an Internet merchant such as a retail 
merchant or an auction merchant. A user, who is at a client computer, may enter 
image information as the search criteria. The image information is used to search all 
the images, or a subset of all the images, in the database to attempt to find items that 
most closely match the image. 

[0006] The image searching may be used as one aspect of the search or as part of a 
multiple criteria search. The client can obtain an image of the item they want, e.g. 
from a digital camera or even a sketch. The search engine then carries out operations 
to find the closest match to the entered image. 

Brief Description of Drawings 

[0007] These and other aspects will now be described in detail with reference to the 

accompanying drawings, wherein:Figure 1 shows a basic system which allows entry of 
image information to search for items to be purchased on the database associated 
with a server;Figure 2 shows a flowchart of operation of entering the image 
information, modifying the image information, and searching the database using the 
results;Figure 3 shows an image of an exemplary item which is difficult to describe 
using text words;Figure 4 shows a user interface for modification of and selection of 
parameters associated with, the image information. 

Detailed Description 

[0008] The present system operates by classifying and identifying images using available 
image detecting techniques. Such image detecting techniques fall into a number of 
different classes. These ways of detecting a closest match image from a plurality of 
images in the database are briefly described, but it should be understood that other 
techniques could also be used. 
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[0009] A first set of techniques forms a class model, usually by making a model of the 
desired image, and using the model as a template. The techniques of this system are 
described in, for example, U.S. Patent No. 5,963,670. 

[0010] Another system, described in, for example U.S. Patent No. 5,781 ,899 measures 
differences between images. Plural pixels in the images may be compared by least 
squares techniques, for example, to find the closest match. 

[001 1] Histograms have been used to compare images. U.S. Patent No. 6,1 81 ,81 7 is 
indicative of the histogram technique. 

[0012] Formation of transforms which represent the image can also be used. The 

transforms have an advantage that artificial learning may be particularly useful in 
processing the transforms. For example, transforms may be operated using a neural 
net system. U.S. Patent No. 6,041 ,147 describes using such transforms. 

[001 3] Another system searches databases of images by forming a context vector or 

feature vector indicative of each image, and selecting among those vectors, according 
to a query. Such a system is described in U.S. Patent No. 6,1 73,275. 

[0014] Any of these systems may be subsets of the others. 

[001 5] In addition, multiple ones of these systems can be combined together to form a 
database search engines such as described in U.S. Patent Nos. 5,91 1 ,1 39; 5,835,667; 
5,751,852; 5,751,286; and others. 



[0016] 



The present application therefore uses one of these existing search engine 
techniques to carry out a new operation, effectively allowing searching a database of 
information; preferably a database that customer decision support, via entry of image 
parts and by searching the database using these image parts. Specific technique is 
described with reference to Figures 1 and 2. Figure 1 illustrates a typical Internet 
transaction via a client/server scenario. A server 1 06 is at a protected space and 
operated by an Internet merchant. The server is preferably associated with an 
electronic commerce site, and enables entry of purchase and/or price information for 
items associated with said search results. That price information may be a 
conventional e-commerce purchase for certain sites. The server in this example may 
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be the eBay server for example, in which case the price information may be an amount 
of a bid on the item. The server may be any other server of any other type on any 
network. While Figure 1 shows a single server, it should be understood that there can 
be multiple networked or mirrored servers, or other multiple servers which are 
configured in some way to accept requests. 

[001 7] Special advantages may be obtained in a system where multiple sellers each 

describe their products, such as on eBay, or in a search engine. In that case, the kinds 
and quantities of products which are sold may be individually described by the various 
sellers, with no consistent description technique being used. 

[001 8] The server 1 00 stores a database of information shown as 1 05. This database of 
information includes pictures 1 08 and text 107. Each item may include either only one 
or more pictures, only one or more text items, or both pictures and text items 
associated with the item. In this first embodiment, the pictures and the text items 
have been entered by multiple different users, e.g. users at different clients. Again, in 
the eBay example, any user at any client can enter any information into the database 
105. 

[001 9] The server 1 00 is connected to the Internet 1 1 0, which may include a flow of 
information carried across any desired channel. A plurality of clients 1 20, 1 22 and 
1 24 are shown connected to the channel 1 1 0. Any number of clients can be 
connected. 

[0020] In this embodiment described herein, information in the database has been 

entered via a number of different sources, with no commonality to the information. Of 
course, however, this may also work when the information has been commonly 
entered by a single source. 

[0021] This system may operate according to the flowchart of Figure 2. 

[0022] At 200, the user enters search criteria that includes image information. The image 
information may be an image or a part of an image. The image may be an image e.g. a 
scanned picture or a hand-drawn sketch, of an item the user wants to find in the 
database. In one embodiment, the client 120 may include a tablet shown as 121 which 
enables sketching pictures. A scanner such as 123 can be used to input electronic 
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images. 

[0023] A sample of the kind of item that would be usable in this system is shown in 

Figure 3. The item shown in Figure 3 is a special kind of cycle with a large front wheel- 
and smaller rear wheel. There is a special name for this contraption, but almost no 
one knows it. If one wanted to find this item on a site that sells a collection of 
merchandise, such as eBay, it might be very difficult to find. However, by using a 
graphical index entry system, closest matches to the images may be found. 

[0024] 210 shows entering additional aspects associated with the image comparison, 
e.g., setting parameters for the search. The user interface for such a selection is 
shown in Figure 4. As described above, both the image, and search text 440 may be 
entered. In addition, different parameters of the search may be specified. The figure 4 
interface defines default or changeable configuration, size, position, orientation and 
lighting of the image information which is entered. 406 shows the image information 
that will be used for the search being "stretched", reoriented, and repositioned in 
order to find closest matches. 

[0025] Figure 4 also allows certain colors to be selected at 41 5. The selection may allow 
returning items having similar shape but different colors. 

[0026] The search may also exclude certain parts, e.g., the user may enter exclusion 
image information of certain elements and request that the search results exclude 
those certain elements. This may be equivalent to the search engine"s entry of a "(- 
)"to indicate items which are to be excluded from the search. For example, a user may 
want to exclude from the search any item which has some specified feature. The 
searching engine will then exclude items which include that specified exclusion image 
information. 

[0027] Parts of the image which are most important may be selected. This may be used to 
wait the searching, so that the search preferentially looks for those more important 
image items. 

[0028] | n Figure 4 user interface, the user sees the image that they are searching for 
as 400. The user has an opportunity to select the most important part 405. The user 
also has an opportunity to enter excluded items shown as 410. in this embodiment, a 
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circle is shown as being excluded. In the user interface as shown, the user may search 
only for triangles which do not include circles as part of the image. 

[0029] Sizing parameters may also be entered as 425; representing the real size of the 
device represented by the image. In the example of Figure 3, the image may return 
matches not only to real size image parts, but also to miniature replicas of such real 
size parts. Real size parameters indicating an overall size; e.g., ranges of inches or 
feet, may be entered. This can allow the search to include or exclude devices of 
certain sizes. 

[0030] Finally, once the search is complete, the server may search the database 1 05 using 
the entered information at 220. The search of the database may use any image 
searching technique, including those described above, including comparison using a 
model, comparisons between image parts, histogram comparison, formation of 
transforms, vectors, or any other technique. 

[0031] Although only a few embodiments have been disclosed in detail above, other 
modifications are possible. 
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